Title: BigQuery: Field
Open Graph Title: BigQuery: Field
X Title: BigQuery: Field
Description: I am encountering the following problem, when uploading a Pandas DataFrame to a partitioned table: Environment details API: BigQuery OS: macOS High Sierra 10.13.6 Python: 3.5.7 Packages: google-api-core==1.11.0 google-api-python-client==...
Open Graph Description: I am encountering the following problem, when uploading a Pandas DataFrame to a partitioned table: Environment details API: BigQuery OS: macOS High Sierra 10.13.6 Python: 3.5.7 Packages: google-api...
X Description: I am encountering the following problem, when uploading a Pandas DataFrame to a partitioned table: Environment details API: BigQuery OS: macOS High Sierra 10.13.6 Python: 3.5.7 Packages: google-api...
Opengraph URL: https://github.com/googleapis/google-cloud-python/issues/8093
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"BigQuery: Field \u003cfield> has changed mode from REQUIRED to NULLABLE","articleBody":"I am encountering the following problem, when uploading a Pandas DataFrame to a partitioned table:\r\n\r\n#### Environment details\r\nAPI: BigQuery\r\nOS: macOS High Sierra 10.13.6\r\nPython: 3.5.7\r\nPackages:\r\n```\r\ngoogle-api-core==1.11.0\r\ngoogle-api-python-client==1.7.8\r\ngoogle-auth==1.6.3\r\ngoogle-auth-httplib2==0.0.3\r\ngoogle-cloud==0.34.0\r\ngoogle-cloud-bigquery==1.12.1\r\ngoogle-cloud-core==1.0.0\r\ngoogle-cloud-dataproc==0.3.1\r\ngoogle-cloud-datastore==1.8.0\r\ngoogle-cloud-storage==1.16.0\r\ngoogle-resumable-media==0.3.2\r\ngoogleapis-common-protos==1.5.10\r\nparquet==1.2\r\n```\r\n\r\n#### Steps to reproduce\r\nCreate a table on BigQuery with the following fields:\r\n- float_value, FLOAT, required\r\n- int_value, INTEGER, required\r\n\r\n#### Reproducible code example (includes creating table)\r\n```python\r\nimport pandas as pd\r\nfrom google.cloud import bigquery\r\n\r\n\r\nPROJECT = \"my-project\"\r\nDATASET = \"my_dataset\"\r\nTABLE = \"my_table\"\r\n\r\n\r\n# My table schema\r\nschema = [\r\n bigquery.SchemaField(\"foo\", \"FLOAT\", mode=\"REQUIRED\"),\r\n bigquery.SchemaField(\"bar\", \"INTEGER\", mode=\"REQUIRED\"),\r\n]\r\n\r\n\r\n# Set everything up\r\nclient = bigquery.Client(PROJECT)\r\ndataset_ref = client.dataset(DATASET)\r\ntable_ref = dataset_ref.table(TABLE)\r\n\r\n\r\n# Delete the table if exists\r\nprint(\"Deleting table if exists...\")\r\nclient.delete_table(table_ref, not_found_ok=True)\r\n\r\n\r\n# Create the table\r\nprint(\"Creating table...\")\r\ntable = bigquery.Table(table_ref, schema=schema)\r\ntable.time_partitioning = bigquery.TimePartitioning(\r\n type_=bigquery.TimePartitioningType.DAY\r\n)\r\ntable = client.create_table(table, exists_ok=True)\r\n\r\nprint(\"Table schema:\")\r\nprint(table.schema)\r\n\r\nprint(\"Table partitioning:\")\r\nprint(table.time_partitioning)\r\n\r\n# Upload data to partition\r\ntable_partition = TABLE + \"$20190522\"\r\ntable_ref = dataset_ref.table(table_partition)\r\n\r\ndf = pd.DataFrame({\"foo\": [1, 2, 3], \"bar\": [2.0, 3.0, 4.0]})\r\nclient.load_table_from_dataframe(df, table_ref).result()\r\n```\r\n\r\n## Output:\r\n```\r\nDeleting table if exists...\r\nCreating table...\r\nTable schema:\r\n[SchemaField('foo', 'FLOAT', 'REQUIRED', None, ()), SchemaField('bar', 'INTEGER', 'REQUIRED', None, ())]\r\nTable partitioning:\r\nTimePartitioning(type=DAY)\r\nTraceback (most recent call last):\r\n File \"\u003cmy-project>/bigquery_failure.py\", line 49, in \u003cmodule>\r\n client.load_table_from_dataframe(df, table_ref).result()\r\n File \"\u003cmy-env>/lib/python3.5/site-packages/google/cloud/bigquery/job.py\", line 732, in result\r\n return super(_AsyncJob, self).result(timeout=timeout)\r\n File \"\u003cmy-env>/lib/python3.5/site-packages/google/api_core/future/polling.py\", line 127, in result\r\n raise self._exception\r\ngoogle.api_core.exceptions.BadRequest:\r\n400 Provided Schema does not match Table my-project:my_dataset.my_table$20190522.\r\nField bar has changed mode from REQUIRED to NULLABLE\r\n\r\nProcess finished with exit code 1\r\n```\r\n\r\n","author":{"url":"https://github.com/timocb","@type":"Person","name":"timocb"},"datePublished":"2019-05-22T12:52:04Z","url":"https://github.com/googleapis/google-cloud-python/issues/8093"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:cb0b0f93-1936-bb78-fe91-bd86d4a2fd60 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | B6C4:2AB11:93F9A1:CF777D:6AFE1CD2 |
| html-safe-nonce | d4e9534e7ee828816001550b9d11b811d43bb9adb1e034c635245178a8e17fd0 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJCNkM0OjJBQjExOjkzRjlBMTpDRjc3N0Q6NkFGRTFDRDIiLCJ2aXNpdG9yX2lkIjoiMzIwNjQ4MzA2NzI3MDk5NTE1NSIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | 3ce9747bfd91a851af6c60a2a3e2bcb89525a5b61a2d32a01f32fa33915405bf |
| hovercard-subject-tag | issue:447118537 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/googleapis/google-cloud-python/8093/issue_layout |
| twitter:image | https://opengraph.githubassets.com/4433194cdbccb71f94b9e2f44577032c59121a7c83804391f666ab51e93f508b/googleapis/google-cloud-python/issues/8093 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/4433194cdbccb71f94b9e2f44577032c59121a7c83804391f666ab51e93f508b/googleapis/google-cloud-python/issues/8093 |
| og:image:alt | I am encountering the following problem, when uploading a Pandas DataFrame to a partitioned table: Environment details API: BigQuery OS: macOS High Sierra 10.13.6 Python: 3.5.7 Packages: google-api... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | timocb |
| hostname | github.com |
| expected-hostname | github.com |
| None | d0525ab1464526bf4532a5551aa348cf100dff75ef808a96fc67a08b1d946aa9 |
| turbo-cache-control | no-preview |
| go-import | github.com/googleapis/google-cloud-python git https://github.com/googleapis/google-cloud-python.git |
| octolytics-dimension-user_id | 16785467 |
| octolytics-dimension-user_login | googleapis |
| octolytics-dimension-repository_id | 16316451 |
| octolytics-dimension-repository_nwo | googleapis/google-cloud-python |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 16316451 |
| octolytics-dimension-repository_network_root_nwo | googleapis/google-cloud-python |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | fef3993a71fa20f7be8e167132fbf9619bc66e12 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width