Title: Allow load_table_from_dataframe to Ignore Extra Schema Fields · Issue #1812 · googleapis/python-bigquery · GitHub
Open Graph Title: Allow load_table_from_dataframe to Ignore Extra Schema Fields · Issue #1812 · googleapis/python-bigquery
X Title: Allow load_table_from_dataframe to Ignore Extra Schema Fields · Issue #1812 · googleapis/python-bigquery
Description: Description: Environment details OS: MacOS Sonoma 14.1.1 Python version: 3.10 google-cloud-bigquery version: 3.17.1 Steps to reproduce Create a BigQuery schema with additional fields not present in the DataFrame. Use load_table_from_data...
Open Graph Description: Description: Environment details OS: MacOS Sonoma 14.1.1 Python version: 3.10 google-cloud-bigquery version: 3.17.1 Steps to reproduce Create a BigQuery schema with additional fields not present in...
X Description: Description: Environment details OS: MacOS Sonoma 14.1.1 Python version: 3.10 google-cloud-bigquery version: 3.17.1 Steps to reproduce Create a BigQuery schema with additional fields not present in...
Opengraph URL: https://github.com/googleapis/python-bigquery/issues/1812
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Allow load_table_from_dataframe to Ignore Extra Schema Fields","articleBody":"Description:\r\n\r\n**Environment details**\r\n\r\nOS: MacOS Sonoma 14.1.1\r\nPython version: 3.10\r\ngoogle-cloud-bigquery version: 3.17.1\r\n\r\n**Steps to reproduce**\r\n\r\nCreate a BigQuery schema with additional fields not present in the DataFrame.\r\nUse load_table_from_dataframe with the defined schema to load data into BigQuery.\r\n\r\n```python\r\nfrom google.cloud import bigquery\r\nfrom google.oauth2 import service_account\r\nimport pandas as pd\r\n\r\ncredentials = service_account.Credentials.from_service_account_file(\r\n 'credentials.json')\r\nclient = bigquery.Client(credentials=credentials, project=credentials.project_id)\r\n\r\njson_data = [\r\n {\r\n \"name\": \"Alice\",\r\n \"age\": 30\r\n },\r\n {\r\n \"name\": \"Bob\",\r\n \"age\": 25\r\n }\r\n]\r\n\r\ndf = pd.DataFrame(json_data)\r\n\r\njob_config = bigquery.LoadJobConfig(schema=[\r\n bigquery.SchemaField(\"name\", \"STRING\"),\r\n bigquery.SchemaField(\"age\", \"INTEGER\"),\r\n bigquery.SchemaField(\"field_not_present\", \"INTEGER\"),\r\n],\r\njob_timeout_ms=5)\r\nload_job = client.load_table_from_dataframe(\r\n df, \"xenon-world-399922.oss.your_table\",\r\n job_config=job_config\r\n)\r\nload_job.result()\r\n```\r\n\r\n**Current behavior**\r\nCurrently, when using load_table_from_dataframe from the Python BigQuery client, if the provided schema contains fields that are not present in the DataFrame, a ValueError is raised:\r\nValueError: bq_schema contains fields not present in dataframe: {'field_not_present'}.\r\n\r\n**Expected behavior**\r\n\r\nIn contrast to the command line behavior when loading JSON data into a BigQuery table, the Python client currently requires a strict match between the DataFrame columns and the provided schema. This behavior can be limiting, as the command line tool does not enforce this match when loading json data.\r\n\r\nI propose that load_table_from_dataframe be enhanced to allow a more flexible schema matching, similar to the command line tool's behavior. Specifically, it should not raise an error if the schema contains additional fields not present in the DataFrame. This would allow for more versatile data loading scenarios where the DataFrame might not always have the complete set of fields defined in the BigQuery table schema.\r\n\r\n**Use case**\r\nThis feature would be particularly useful in scenarios where the DataFrame is dynamically generated and might not always contain the full set of fields as per the BigQuery schema. Allowing the function to ignore extra schema fields would enable more flexible and robust data loading operations.","author":{"url":"https://github.com/ArnoldHueteG","@type":"Person","name":"ArnoldHueteG"},"datePublished":"2024-02-07T04:30:05Z","url":"https://github.com/googleapis/python-bigquery/issues/1812"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:6a6490cd-a579-c226-c419-008c6b01e43f |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | 8CEE:245A7:1244262:185E90F:6AFEAB93 |
| html-safe-nonce | 393398e825fe185f8f7e026365a79cc3db97e3a5825ce1e5ab2987fedb1d9359 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI4Q0VFOjI0NUE3OjEyNDQyNjI6MTg1RTkwRjo2QUZFQUI5MyIsInZpc2l0b3JfaWQiOiIzNDQwMTYwODM0NzA4MjIwODE5IiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | 6e650d99374927e4be58a077a13c0b5604e98e432933c61338fba88100738956 |
| hovercard-subject-tag | issue:2122164825 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/googleapis/python-bigquery/1812/issue_layout |
| twitter:image | https://opengraph.githubassets.com/b2e66e401434f1b3736e3acc30f55fbafa3b0710c13d7948d9c445bcd81e9198/googleapis/python-bigquery/issues/1812 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/b2e66e401434f1b3736e3acc30f55fbafa3b0710c13d7948d9c445bcd81e9198/googleapis/python-bigquery/issues/1812 |
| og:image:alt | Description: Environment details OS: MacOS Sonoma 14.1.1 Python version: 3.10 google-cloud-bigquery version: 3.17.1 Steps to reproduce Create a BigQuery schema with additional fields not present in... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | ArnoldHueteG |
| hostname | github.com |
| expected-hostname | github.com |
| None | 95b2753c978240785d6cca00783c00a371b2d7d9879bb32e305170a1b19bd28e |
| turbo-cache-control | no-preview |
| go-import | github.com/googleapis/python-bigquery git https://github.com/googleapis/python-bigquery.git |
| octolytics-dimension-user_id | 16785467 |
| octolytics-dimension-user_login | googleapis |
| octolytics-dimension-repository_id | 226992475 |
| octolytics-dimension-repository_nwo | googleapis/python-bigquery |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 226992475 |
| octolytics-dimension-repository_network_root_nwo | googleapis/python-bigquery |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 3f25273206cd841171bca0002261ae4c9757d934 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width