Title: `Client.query()` sometimes raises 404, despite the query job being successfully started · Issue #2134 · googleapis/python-bigquery · GitHub
Open Graph Title: `Client.query()` sometimes raises 404, despite the query job being successfully started · Issue #2134 · googleapis/python-bigquery
X Title: `Client.query()` sometimes raises 404, despite the query job being successfully started · Issue #2134 · googleapis/python-bigquery
Description: Environment details OS type and version: Google Colab Python version: python --version 3.11 pip version: pip --version N/A google-cloud-bigquery version: pip show google-cloud-bigquery 3.25.0 Steps to reproduce It's a bit difficult to re...
Open Graph Description: Environment details OS type and version: Google Colab Python version: python --version 3.11 pip version: pip --version N/A google-cloud-bigquery version: pip show google-cloud-bigquery 3.25.0 Steps...
X Description: Environment details OS type and version: Google Colab Python version: python --version 3.11 pip version: pip --version N/A google-cloud-bigquery version: pip show google-cloud-bigquery 3.25.0 Steps...
Opengraph URL: https://github.com/googleapis/python-bigquery/issues/2134
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"`Client.query()` sometimes raises 404, despite the query job being successfully started","articleBody":"#### Environment details\n\n - OS type and version: Google Colab\n - Python version: `python --version` 3.11\n - pip version: `pip --version` N/A\n - `google-cloud-bigquery` version: `pip show google-cloud-bigquery` 3.25.0\n\n#### Steps to reproduce\n\nIt's a bit difficult to reproduce, but it seems to happen most often with creating a vector index via [CREATE VECTOR INDEX DDL](https://cloud.google.com/bigquery/docs/vector-index#create_a_vector_index) as generated by [bigframes.bigquery.create_vector_index](https://cloud.google.com/python/docs/reference/bigframes/latest/bigframes.bigquery#bigframes_bigquery_create_vector_index). \n\n#### Code example\n\n\u003cdetails>\u003csummary>Long Python sample, sorry\u003c/summary>\n\n```python\nPROJECT_ID = \"my-project\" # @param {type:\"string\"}\nDATASET_ID = \"my_dataset\" # @param {type:\"string\"}\nCONNECTION_ID = \"bqml_llm_conn\" # @param {type:\"string\"}\n\n# Set the project id\n! gcloud config set project {PROJECT_ID}\n\nREGION = \"US\" # @param {type: \"string\"}\n\n## Text embedding variables\n\nTEXT_TABLE_ID = \"PATENT_TABLE_BF\"\nTEXT_MODEL_ID = \"llm_embedding_model\"\nTEXT_EMBEDDING_TABLE_ID = \"patent_embedding_BF_test\"\nTEXT_EMBEDDING_TABLE_ID2 = \"patent_embedding\"\nTEXT_MODEL_ENDPOINT = \"text-embedding-004\"\nTEXT_VECTOR_INDEX_ID = \"patent_index_BF\"\n\nimport bigframes.pandas as bf\nimport bigframes.ml as bf_ml\nimport bigframes.bigquery as bf_bq\nimport bigframes.ml.llm as bf_llm\n\n\nfrom google.cloud import bigquery\nfrom google.cloud import storage\n\n# Construct a BigQuery client object.\nclient = bigquery.Client()\n\nimport pandas as pd\nfrom IPython.display import Image, display\nfrom PIL import Image as PILImage\nimport io\n\nbf.options.bigquery.project = PROJECT_ID\nbf.options.bigquery.location = REGION\nbf.options.bigquery.ordering_mode = \"partial\"\n\ntext_model = bf_llm.TextEmbeddingGenerator() ## which endpoint this uses??\n\npublications = bf.read_gbq('patents-public-data.google_patents_research.publications')\nkeep = (publications.embedding_v1.str.len() > 0) & (publications.title.str.len() > 0) & (publications.abstract.str.len() > 0)\npublications = publications[[\"publication_number\", \"title\", \"abstract\"]].rename(columns={'abstract': 'content'})\n\n## creating embeddings for 10000 rows only for testing, change in the line below for embedding higher number of rows\npublications_subset = publications[publications[\"content\"].str.len() > 30].peek(10000)\n## publications_subset = publications_subset[[\"publication_number\", \"title\", \"abstract\"]].rename(columns={'abstract': 'content'})\n\n\nembedding = text_model.predict(publications_subset)[[\"publication_number\", \"title\", \"content\", \"ml_generate_embedding_result\",\"ml_generate_embedding_status\"]]\n\n## filter out rows where the embedding generation failed. the embedding status value is empty if the embedding generation was successful\nembedding = embedding[~embedding[\"ml_generate_embedding_status\"].isnull()]\n\n# store embeddings in a BQ table\nembedding.to_gbq(f\"{DATASET_ID}.{TEXT_EMBEDDING_TABLE_ID}\", if_exists='replace')\n\n## python code to create index\nbf_bq.create_vector_index(\n table_id = f\"{DATASET_ID}.{TEXT_EMBEDDING_TABLE_ID}\",\n column_name = \"ml_generate_embedding_result\",\n replace= True,\n index_name = \"bf_python_index\",\n distance_type=\"cosine\",\n index_type= \"ivf\"\n)\n```\n\n\u003c/details>\n\nUnfortunately, even this is not a guarantee, but it does seem pretty reproducible on one of the projects my colleague is using.\n\n#### Stack trace\n\n```\n/usr/local/lib/python3.10/dist-packages/bigframes/session/_io/bigquery/__init__.py in start_query_with_client(bq_client, sql, job_config, max_results, timeout, api_name, metrics)\n 229 \n 230 try:\n--> 231 query_job = bq_client.query(sql, job_config=job_config, timeout=timeout)\n 232 except google.api_core.exceptions.Forbidden as ex:\n 233 if \"Drive credentials\" in ex.message:\n\n/usr/local/lib/python3.10/dist-packages/google/cloud/bigquery/client.py in query(self, query, job_config, job_id, job_id_prefix, location, project, retry, timeout, job_retry, api_method)\n 3490 )\n 3491 elif api_method == enums.QueryApiMethod.INSERT:\n-> 3492 return _job_helpers.query_jobs_insert(\n 3493 self,\n 3494 query,\n\n/usr/local/lib/python3.10/dist-packages/google/cloud/bigquery/_job_helpers.py in query_jobs_insert(client, query, job_config, job_id, job_id_prefix, location, project, retry, timeout, job_retry)\n 157 return query_job\n 158 \n--> 159 future = do_query()\n 160 # The future might be in a failed state now, but if it's\n 161 # unrecoverable, we'll find out when we ask for it's result, at which\n\n/usr/local/lib/python3.10/dist-packages/google/cloud/bigquery/_job_helpers.py in do_query()\n 143 \n 144 try:\n--> 145 query_job = client.get_job(\n 146 job_id,\n 147 project=project,\n\n/usr/local/lib/python3.10/dist-packages/google/cloud/bigquery/client.py in get_job(self, job_id, project, location, retry, timeout)\n 2199 span_attributes = {\"path\": path, \"job_id\": job_id, \"location\": location}\n 2200 \n-> 2201 resource = self._call_api(\n 2202 retry,\n 2203 span_name=\"BigQuery.getJob\",\n\n/usr/local/lib/python3.10/dist-packages/google/cloud/bigquery/client.py in _call_api(self, retry, span_name, span_attributes, job_ref, headers, **kwargs)\n 831 name=span_name, attributes=span_attributes, client=self, job_ref=job_ref\n 832 ):\n--> 833 return call()\n 834 \n 835 return call()\n\n/usr/local/lib/python3.10/dist-packages/google/api_core/retry/retry_unary.py in retry_wrapped_func(*args, **kwargs)\n 291 self._initial, self._maximum, multiplier=self._multiplier\n 292 )\n--> 293 return retry_target(\n 294 target,\n 295 self._predicate,\n\n/usr/local/lib/python3.10/dist-packages/google/api_core/retry/retry_unary.py in retry_target(target, predicate, sleep_generator, timeout, on_error, exception_factory, **kwargs)\n 151 except Exception as exc:\n 152 # defer to shared logic for handling errors\n--> 153 _retry_error_helper(\n 154 exc,\n 155 deadline,\n\n/usr/local/lib/python3.10/dist-packages/google/api_core/retry/retry_base.py in _retry_error_helper(exc, deadline, next_sleep, error_list, predicate_fn, on_error_fn, exc_factory_fn, original_timeout)\n 210 original_timeout,\n 211 )\n--> 212 raise final_exc from source_exc\n 213 if on_error_fn is not None:\n 214 on_error_fn(exc)\n\n/usr/local/lib/python3.10/dist-packages/google/api_core/retry/retry_unary.py in retry_target(target, predicate, sleep_generator, timeout, on_error, exception_factory, **kwargs)\n 142 for sleep in sleep_generator:\n 143 try:\n--> 144 result = target()\n 145 if inspect.isawaitable(result):\n 146 warnings.warn(_ASYNC_RETRY_WARNING)\n\n/usr/local/lib/python3.10/dist-packages/google/cloud/_http/__init__.py in api_request(self, method, path, query_params, data, content_type, headers, api_base_url, api_version, expect_json, _target_object, timeout, extra_api_info)\n 492 \n 493 if not 200 \u003c= response.status_code \u003c 300:\n--> 494 raise exceptions.from_http_response(response)\n 495 \n 496 if expect_json and response.content:\n\nNotFound: 404 GET https://bigquery.googleapis.com/bigquery/v2/projects/my-project/jobs/c540a00e-05ed-4caf-b76c-b23d16b9617f?projection=full&location=US&prettyPrint=false: Not found: Job my-project:US.c540a00e-05ed-4caf-b76c-b23d16b9617f\n```\n\nNote that it is getting a 404 here: https://github.com/googleapis/python-bigquery/blob/b03a2afabde7f42be45f62fabd3dc0e6a9a493e1/google/cloud/bigquery/_job_helpers.py#L145\n\nThis is after a Conflict exception, meaning the job was created successfully, the retry failed because it already exists, but then getting that already existing job failed with 404. Something funky is going on with the networking I think for this case to happen, but it should be possible to simulate in unit tests, at least.\n","author":{"url":"https://github.com/tswast","@type":"Person","name":"tswast"},"datePublished":"2025-02-18T20:05:43Z","url":"https://github.com/googleapis/python-bigquery/issues/2134"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:b5b1416b-bc9f-83eb-72df-2ac28441aafb |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | 894C:245A7:10D11C4:1661287:6AFEA47D |
| html-safe-nonce | 792e3dda61fa60d184d2d43a71cc65d070d14c508d800ca72ad952a5c8468920 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI4OTRDOjI0NUE3OjEwRDExQzQ6MTY2MTI4Nzo2QUZFQTQ3RCIsInZpc2l0b3JfaWQiOiI4OTY0MDgwMzM2MDk0Nzk4OTczIiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | e06f465e5f82236009875049fa267ed14602c31dd08e459930cd5290fb0ba9ee |
| hovercard-subject-tag | issue:2861530571 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/googleapis/python-bigquery/2134/issue_layout |
| twitter:image | https://opengraph.githubassets.com/6c9569024b937ddf40321af990429c987a6198696c50364da033193254cdaab2/googleapis/python-bigquery/issues/2134 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/6c9569024b937ddf40321af990429c987a6198696c50364da033193254cdaab2/googleapis/python-bigquery/issues/2134 |
| og:image:alt | Environment details OS type and version: Google Colab Python version: python --version 3.11 pip version: pip --version N/A google-cloud-bigquery version: pip show google-cloud-bigquery 3.25.0 Steps... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | tswast |
| hostname | github.com |
| expected-hostname | github.com |
| None | 95b2753c978240785d6cca00783c00a371b2d7d9879bb32e305170a1b19bd28e |
| turbo-cache-control | no-preview |
| go-import | github.com/googleapis/python-bigquery git https://github.com/googleapis/python-bigquery.git |
| octolytics-dimension-user_id | 16785467 |
| octolytics-dimension-user_login | googleapis |
| octolytics-dimension-repository_id | 226992475 |
| octolytics-dimension-repository_nwo | googleapis/python-bigquery |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 226992475 |
| octolytics-dimension-repository_network_root_nwo | googleapis/python-bigquery |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 3f25273206cd841171bca0002261ae4c9757d934 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width