Title: BigQuery: Field 'bar' is specified as REPEATED in provided schema which does not match REQUIRED as specified in the file. · Issue #17 · googleapis/python-bigquery · GitHub
Open Graph Title: BigQuery: Field 'bar' is specified as REPEATED in provided schema which does not match REQUIRED as specified in the file. · Issue #17 · googleapis/python-bigquery
X Title: BigQuery: Field 'bar' is specified as REPEATED in provided schema which does not match REQUIRED as specified in the file. · Issue #17 · googleapis/python-bigquery
Description: I have an issue where it is not possible to upload a Pandas DataFrame with a repeated field to BigQuery. It is very much related to an issue I've had earlier: googleapis/google-cloud-python#8093 Since that has been resolved (by being abl...
Open Graph Description: I have an issue where it is not possible to upload a Pandas DataFrame with a repeated field to BigQuery. It is very much related to an issue I've had earlier: googleapis/google-cloud-python#8093 Si...
X Description: I have an issue where it is not possible to upload a Pandas DataFrame with a repeated field to BigQuery. It is very much related to an issue I've had earlier: googleapis/google-cloud-python#809...
Opengraph URL: https://github.com/googleapis/python-bigquery/issues/17
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"BigQuery: Field 'bar' is specified as REPEATED in provided schema which does not match REQUIRED as specified in the file.","articleBody":"I have an issue where it is not possible to upload a Pandas DataFrame with a repeated field to BigQuery. It is very much related to an issue I've had earlier: https://github.com/googleapis/google-cloud-python/issues/8093\r\n\r\nSince that has been resolved (by being able to specify the schema), I've created a separate issue. I also couldn't find issues related to REPEATED fields.\r\n\r\n#### Environment details\r\nMac OS X 10.14.5\r\nPython 3.6.8\r\n\r\nPackages:\r\n```\r\ngoogle-api-core==1.14.2\r\ngoogle-auth==1.6.3\r\ngoogle-cloud-bigquery==1.19.0\r\ngoogle-cloud-core==1.0.3\r\ngoogle-cloud-iam==0.2.1\r\ngoogle-cloud-logging==1.12.1\r\ngoogle-resumable-media==0.3.3\r\ngoogleapis-common-protos==1.6.0\r\n```\r\n\r\n#### Steps to reproduce\r\n1. Have a table with a REPEATED field\r\n2. Upload a Pandas DataFrame with a repeated field to this table\r\n3. Error\r\n\r\nAlso:\r\n- Getting the schema from BigQuery and using that in the `JobConfig` doesn't change the error.\r\n\r\n#### Code example\r\n```python\r\nimport pandas as pd\r\nfrom google.cloud import bigquery\r\n\r\n\r\nPROJECT = \"MY-PROJECT\"\r\nDATASET = \"MY_DATASET\"\r\nTABLE = \"MY_TABLE\"\r\n\r\n\r\n# My table schema\r\nschema = [\r\n bigquery.SchemaField(\"foo\", \"INTEGER\", mode=\"REQUIRED\"),\r\n bigquery.SchemaField(\"bar\", \"FLOAT\", mode=\"REPEATED\"),\r\n]\r\n\r\n\r\n# Set everything up\r\nclient = bigquery.Client(PROJECT)\r\ndataset_ref = client.dataset(DATASET)\r\ntable_ref = dataset_ref.table(TABLE)\r\n\r\n\r\n# Delete the table if exists\r\nprint(\"Deleting table if exists...\")\r\nclient.delete_table(table_ref, not_found_ok=True)\r\n\r\n\r\n# Create the table\r\nprint(\"Creating table...\")\r\ntable = bigquery.Table(table_ref, schema=schema)\r\ntable.time_partitioning = bigquery.TimePartitioning(\r\n type_=bigquery.TimePartitioningType.DAY\r\n)\r\ntable = client.create_table(table, exists_ok=True)\r\n\r\nprint(\"Table schema:\")\r\nprint(table.schema)\r\n\r\nprint(\"Table partitioning:\")\r\nprint(table.time_partitioning)\r\n\r\n# Upload data to partition\r\ntable_partition = TABLE + \"$20190522\"\r\ntable_ref = dataset_ref.table(table_partition)\r\n\r\ndf = pd.DataFrame({\"foo\": [1, 2, 3], \"bar\": [[2.0, 3.0], [3.0, 4.0], [4.0, 5.0]]})\r\n\r\njob_config = bigquery.LoadJobConfig(schema=schema)\r\nclient.load_table_from_dataframe(df, table_ref, job_config=job_config).result()\r\n```\r\n\r\n#### Stack trace\r\n```\r\nTraceback (most recent call last):\r\n File \"test.py\", line 51, in \u003cmodule>\r\n client.load_table_from_dataframe(df, table_ref, job_config=job_config).result()\r\n File \"google/cloud/bigquery/job.py\", line 734, in result\r\n return super(_AsyncJob, self).result(timeout=timeout)\r\n File \"google/api_core/future/polling.py\", line 127, in result\r\n raise self._exception\r\ngoogle.api_core.exceptions.BadRequest: 400 Error while reading data, error message: \r\n\r\nProvided schema is not compatible with the file 'prod-scotty-******'. \r\n\r\nField 'bar' is specified as REPEATED in provided schema \r\nwhich does not match REQUIRED as specified in the file.\r\n```\r\n","author":{"url":"https://github.com/timocb","@type":"Person","name":"timocb"},"datePublished":"2019-09-11T08:51:38Z","url":"https://github.com/googleapis/python-bigquery/issues/17"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:b3bd5a63-51ce-722d-678d-0b2d2d03440e |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | C27E:39556A:E4BEAE:136015A:6AFE7C70 |
| html-safe-nonce | ec80ce929db10e84c34861f7b1998974c7b9b7f6d8f1b43272a5e2f96bad3860 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJDMjdFOjM5NTU2QTpFNEJFQUU6MTM2MDE1QTo2QUZFN0M3MCIsInZpc2l0b3JfaWQiOiI2OTk3ODcxMTMwMDU2NzUyMjQwIiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | f9dc0d7fe01a0a6ae258960ad579efd89e9d4125f476986edf512a896c0f32b5 |
| hovercard-subject-tag | issue:559705632 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/googleapis/python-bigquery/17/issue_layout |
| twitter:image | https://opengraph.githubassets.com/dc2df4533bc405c3ac57d9afc967a43c31b0e8b0a688a8aa46743874d5003b9d/googleapis/python-bigquery/issues/17 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/dc2df4533bc405c3ac57d9afc967a43c31b0e8b0a688a8aa46743874d5003b9d/googleapis/python-bigquery/issues/17 |
| og:image:alt | I have an issue where it is not possible to upload a Pandas DataFrame with a repeated field to BigQuery. It is very much related to an issue I've had earlier: googleapis/google-cloud-python#8093 Si... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | timocb |
| hostname | github.com |
| expected-hostname | github.com |
| None | b3c0f155eafc27e60db6e254d1ffff0e072e9ab3b4018a5ee518797c19d89f86 |
| turbo-cache-control | no-preview |
| go-import | github.com/googleapis/python-bigquery git https://github.com/googleapis/python-bigquery.git |
| octolytics-dimension-user_id | 16785467 |
| octolytics-dimension-user_login | googleapis |
| octolytics-dimension-repository_id | 226992475 |
| octolytics-dimension-repository_nwo | googleapis/python-bigquery |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 226992475 |
| octolytics-dimension-repository_network_root_nwo | googleapis/python-bigquery |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 4a10310af51a1bfffbedc4fd81cb582acf0cb0ff |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width